Posterior Simulation for Feed Forward Neural Network Models

1 Motivation We are interested in Bayesian inference and prediction with feed-forward neural network models (FFNN's), speciically those with one hidden layer with M hidden nodes, p input nodes, 1 output node and logistic activation functions: we try to predict a variable y in terms of p variables x = (x 1 ; :::; x p), with regression function y(x) = P M i=1 j (j x) where (z) = exp(z) 1+exp(z). These and other neural network models are the central theme of recent research. Statistical introductions may be seen in Cheng and Titterington (1994) and Ripley (1993). Neural networks are typically presented as black box models to deal with nonlinear features in problems like regression, forecasting and classiication. Incorporating prior knowledge in those models should enhance their performance. This naturally begs for a Bayesian approach, see Buntine and Weigend for some views. Among other advantages, the Bayesian approach allows for coherent incorporation of all uncertainties, hence permitting coherent procedures to network architecture choice, one of the main problems in NN research. However, it leads to diicult computational problems, stemming from non-normality and multimodality of posterior distributions, which hinder the use of methods like Laplace integration, Gaussian quadrature and Monte Carlo importance sampling. Multimodality issues have predated discussions in neu-ral network research, see e.g. Ripley (1993), and are relevant as well for mixture models, see West, M uller and Escobar (1994) and Crawford (1994), of which FFNN's are a special case. There are three main reasons for multimodality of posterior models in FFNN's. The rst one is symmetries due to relabeling; we mitigate this problem introducing appropriate inequality constraints among parameters. The second, and most worrisome, is the inclusion of several copies of the same term, in our case, terms with the same vector. Node duplication may be